A presentation for Research Data at the Edge, Day One of Duke Research Computing Symposium
Why?
- The Web has lots of stuff
- frontier beyond currated datasets
- stuff is wrapped in HTML
- The HTML is transported over HTTP but composed for h2m consumption
- Intellectual Property Rights bear serious consideration
API
Aplication Program Interface
- Built for machine-to-machine interactions
- Instructions for programs
Client / Server
- Make [R] interface with the web, same as h2m but now m2m
Human Simulation
- Human version
- Person uses Web Client
- Person enters a URL
- client & server negotiate

- Information is sent back in wrapped HTML
- Web Browser parses the HTML
JSON
- Javascript Object Notation is a languague-indpendent data format
- Currently the most common data data format for asynchronous client/server communication format
- Consists of key-value pairs
# from https://en.wikipedia.org/wiki/JSON
{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 25,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "office",
"number": "646 555-4567"
},
{
"type": "mobile",
"number": "123 456-7890"
}
],
"children": [],
"spouse": null
}
LS0tDQp0aXRsZTogIlVzaW5nIFIgdG8gb3JjaGVzdHJhdGUgQVBJcyINCmF1dGhvcjogIkpvaG4gTGl0dGxlIg0KZGF0ZTogJ2ByIFN5cy5EYXRlKClgJw0Kb3V0cHV0Og0KICBzbGlkeV9wcmVzZW50YXRpb246IGRlZmF1bHQNCiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdA0KLS0tDQpBIHByZXNlbnRhdGlvbiBmb3IgW1Jlc2VhcmNoIERhdGEgYXQgdGhlIEVkZ2VdKGh0dHA6Ly9saWJyYXJ5LmR1a2UuZWR1L2VkZ2UvZXZlbnRzL3JjMTcpLCBEYXkgT25lIG9mIFtEdWtlIFJlc2VhcmNoIENvbXB1dGluZyBTeW1wb3NpdW1dKGh0dHBzOi8vcmMuZHVrZS5lZHUvc3ltcG9zaXVtLTIwMTcvKQ0KDQojIyBPdXRsaW5lDQoNCiogQVBJDQoqIEpTT04NCiogUiAvIFJTdHVkaW8NCg0KIyMgV2h5Pw0KPCEtLSBOQVNBIGFuaW1hdGVkIEdJRiAvLy8gIGh0dHA6Ly9pLmdpcGh5LmNvbS9sMkpodDRsSWZFUWZKM3pqMi5naWYgICAgLS0+IA0KPCEtLSAgZ29vZCBodW1hbiBoYW5kc2hha2UgLy8vICBodHRwOi8vZ2lwaHkuY29tL2dpZnMvdGhvbWFzLVUyWGJvUnVOODlJZGkgLS0+DQo8IS0tIGFmdGVyIHRoZSByZXNlYXJjaCBoYW5kc2hha2UgaXMgY29tcGxldGUgLy8vIGh0dHA6Ly9naXBoeS5jb20vZ2lmcy84MHMtMTk4MHMtdGhvbWFzLWRvbGJ5LXdDS21CZDdvTnRBNGcgIC0tPiANCjwhLS0gdGhlIGNvbmZ1c2lvbiBvZiB0aGUgbTJtIGhhbmRzaGFrZSAvLy8gICBodHRwOi8vZ2lwaHkuY29tL2dpZnMvdGhvbWFzLU1qa0NZak00Nk5yck8gLS0+DQoqIFRoZSBXZWIgaGFzIGxvdHMgb2Ygc3R1ZmYNCiAgICArIGZyb250aWVyIGJleW9uZCBjdXJyYXRlZCBkYXRhc2V0cw0KICAgICsgc3R1ZmYgaXMgd3JhcHBlZCBpbiBIVE1MDQogICAgKyBUaGUgSFRNTCBpcyB0cmFuc3BvcnRlZCBvdmVyIEhUVFAgYnV0IGNvbXBvc2VkIGZvciBoMm0gY29uc3VtcHRpb24NCiAgICArIEludGVsbGVjdHVhbCBQcm9wZXJ0eSBSaWdodHMgYmVhciBzZXJpb3VzIGNvbnNpZGVyYXRpb24NCg0KIyMgQVBJDQpBcGxpY2F0aW9uIFByb2dyYW0gSW50ZXJmYWNlIA0KDQoqIEJ1aWx0IGZvciBtYWNoaW5lLXRvLW1hY2hpbmUgaW50ZXJhY3Rpb25zDQoqIEluc3RydWN0aW9ucyBmb3IgcHJvZ3JhbXMNCg0KLS0tICAgIA0KDQojIyMgQ2xpZW50IC8gU2VydmVyIA0KPCEtLSBodHRwczovL3BpeGFiYXkuY29tL2VuL2NsaWVudC1zZXJ2ZXItbmV0d29ya2luZy1sYXB0b3AtMzQxNDIwLyAtLT4NCg0KIVtdKGh0dHBzOi8vdXBsb2FkLndpa2ltZWRpYS5vcmcvd2lraXBlZGlhL2NvbW1vbnMvdGh1bWIvYy9jOS9DbGllbnQtc2VydmVyLW1vZGVsLnN2Zy81MDBweC1DbGllbnQtc2VydmVyLW1vZGVsLnN2Zy5wbmcpIA0KDQoqIE1ha2UgW1JdIGludGVyZmFjZSB3aXRoIHRoZSB3ZWIsIHNhbWUgYXMgaDJtIGJ1dCBub3cgbTJtDQoNCi0tLSAgDQoNCiMjIyBIdW1hbiBTaW11bGF0aW9uDQoqIEh1bWFuIHZlcnNpb24NCiAgICArIFBlcnNvbiB1c2VzIFdlYiBDbGllbnQNCiAgICArIFBlcnNvbiBlbnRlcnMgYSBVUkw8YnI+DQogICAgIVtdKGh0dHBzOi8vdXBsb2FkLndpa2ltZWRpYS5vcmcvd2lraXBlZGlhL2NvbW1vbnMvZC9kNi9Vbmlmb3JtX1Jlc291cmNlX0xvY2F0b3JfJTI4VVJMJTI5X2V4YW1wbGUuUE5HKQ0KICAgIDwhLS0gaHR0cHM6Ly9jb21tb25zLndpa2ltZWRpYS5vcmcvd2lraS9GaWxlOlVuaWZvcm1fUmVzb3VyY2VfTG9jYXRvcl8oVVJMKV9leGFtcGxlLlBORyAtLT4NCiAgICArIGNsaWVudCAmIHNlcnZlciBuZWdvdGlhdGU8YnI+IA0KICAgICFbZHJhbWF0aXphdGlvbjogZ29vZCBoYW5kc2hha2VdKGh0dHA6Ly9pLmdpcGh5LmNvbS9VMlhib1J1Tjg5SWRpLmdpZikgDQogICAgKyBJbmZvcm1hdGlvbiBpcyBzZW50IGJhY2sgaW4gd3JhcHBlZCBIVE1MDQogICAgKyBXZWIgQnJvd3NlciBwYXJzZXMgdGhlIEhUTUwgDQogICAgPCEtLSBodHRwczovL2NvbW1vbnMud2lraW1lZGlhLm9yZy93aWtpL0ZpbGU6SFRNTC5zdmcgLS0+DQoNCg0KDQoNCiMjIG0ybSAtLSBkZXZlbG9wbWVudA0KDQoNCiFbZHJhbWF0aXphdGlvbjogY29uZnVzZWQgYWJvdXQgdGhlIHByb3RvY29sXShodHRwOi8vaS5naXBoeS5jb20vTWprQ1lqTTQ2TnJyTy5naWYpDQogICAgDQojIyBKU09ODQoNCiogW0phdmFzY3JpcHQgT2JqZWN0IE5vdGF0aW9uXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9KU09OKSBpcyBhIGxhbmd1YWd1ZS1pbmRwZW5kZW50IGRhdGEgZm9ybWF0DQoqIEN1cnJlbnRseSB0aGUgbW9zdCBjb21tb24gZGF0YSBkYXRhIGZvcm1hdCBmb3IgYXN5bmNocm9ub3VzIGNsaWVudC9zZXJ2ZXIgY29tbXVuaWNhdGlvbiBmb3JtYXQNCiogQ29uc2lzdHMgb2Yga2V5LXZhbHVlIHBhaXJzDQoNCjwhLS0gaHR0cDovL2kudmltZW9jZG4uY29tL3ZpZGVvLzU0MTkzNTgxNl8xMjgweDcyMC5qcGcgLS0+DQo8IS0tIFZpbWVvIG9uIFdoYXQgaXMgSlNPTiAvLyBodHRwczovL3ZpbWVvLmNvbS8xNDQxNjIxMDIgLS0+DQoNCg0KYGBge2pzb24gZXhhbXBsZX0NCiMgZnJvbSBodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9KU09ODQp7DQogICJmaXJzdE5hbWUiOiAiSm9obiIsDQogICJsYXN0TmFtZSI6ICJTbWl0aCIsDQogICJpc0FsaXZlIjogdHJ1ZSwNCiAgImFnZSI6IDI1LA0KICAiYWRkcmVzcyI6IHsNCiAgICAic3RyZWV0QWRkcmVzcyI6ICIyMSAybmQgU3RyZWV0IiwNCiAgICAiY2l0eSI6ICJOZXcgWW9yayIsDQogICAgInN0YXRlIjogIk5ZIiwNCiAgICAicG9zdGFsQ29kZSI6ICIxMDAyMS0zMTAwIg0KICB9LA0KICAicGhvbmVOdW1iZXJzIjogWw0KICAgIHsNCiAgICAgICJ0eXBlIjogImhvbWUiLA0KICAgICAgIm51bWJlciI6ICIyMTIgNTU1LTEyMzQiDQogICAgfSwNCiAgICB7DQogICAgICAidHlwZSI6ICJvZmZpY2UiLA0KICAgICAgIm51bWJlciI6ICI2NDYgNTU1LTQ1NjciDQogICAgfSwNCiAgICB7DQogICAgICAidHlwZSI6ICJtb2JpbGUiLA0KICAgICAgIm51bWJlciI6ICIxMjMgNDU2LTc4OTAiDQogICAgfQ0KICBdLA0KICAiY2hpbGRyZW4iOiBbXSwNCiAgInNwb3VzZSI6IG51bGwNCn0NCmBgYA0KDQoNCiMjIEV4YW1wbGUNCg0KIyMjIE9NREIgYXBpIA0KbGlrZSBodHRwOi8vaW1kYi5jb20vDQoNCiMjIFIgUGFja2FnZXMgLS0gUmVsYXRlZA0KDQoqUGVvcGxlIHdobyB1c2UgSlNPTmxpdGUgYWxzbyB1c2UuLi4qDQoNCiogaHR0ciAtLSBjYWxscyBKU09ObGl0ZSBpbiBzZXJ2aWNlIHRvIG1ham9yIGdvYWwgb2YgbWFuYWdpbmcgSFRUUCANCiogcnZlc3QgLS0gIHVzZWQgZm9yIGh0bWwgcGFyc2luZw0KDQoNCg0K